Electronic apparatus, method for controlling thereof and the computer readable recording medium

ABSTRACT

An electronic device is provided. The electronic device includes a light emitting unit configured to alternately emit light having a preset pattern to an eye area of a user; an inputter configured to receive a plurality of images which capture an eye area of the user; and a processor configured to detect a user&#39;s gaze using a first image that is captured at the time of irradiating a preset first pattern and a second image that is captured at the time of irradiating a preset second pattern, and the processor detects a user&#39;s gaze by extracting reflection points of each of the first image and the second image and using remaining reflection points excluding reflection points at a same position from among the extracted reflection points.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2017-0113650, filed in the KoreanIntellectual Property Office on Sep. 6, 2017, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

Aspects of the example embodiments relate generally to an electronicdevice, a control method thereof, and a computer readable recordingmedium and, more particularly, to an electronic device for detecting auser's gaze at outdoor environments, a control method thereof, and acomputer readable recording medium.

2. Description of Related Art

Recently, there is a need for gaze tracking technology that isapplicable to an outdoor environment as automobile, advertisement,marketing fields and so on require gaze tracking. In order to performgaze tracking in an outdoor environment, it is most important toaccurately detect the feature points of a user's eyes even under anexternal lighting environment. However, most of the conventionaltechniques are under the assumption of an indoor environment, and notmany studies have been made regarding the feature point detectiontechnique for the outdoor environment. Recently, adaptive techniques tothe outdoor environment can be classified into a technique using asingle image and a technique using a multi-image which captures user'seyes. However, the technique using a single image requires priorinformation on the feature points of the eye as well as priorinformation on the pattern of the infrared emitter, and the performancebecomes unstable when the influence of external lighting is strong. Inaddition, the technique using multi-image requires complicated hardwaresuch as requiring a high-speed camera.

Accordingly, there is a need for a technique for accurately detectingfeature points of eyes in an outdoor environment using a low-costgeneral camera without prior knowledge of the pattern of an infraredradiator.

SUMMARY

An aspect of the example embodiments relates to an electronic devicecapable of accurately detecting feature points to detect a user's gazeeven under various external lighting, a control method thereof, and acomputer readable recording medium.

According to an exemplary embodiment, an electronic device includes alight emitting unit configured to alternately emit light having a presetpattern to an eye area of a user; an inputter configured to receive aplurality of images which capture an eye area of the user; and aprocessor configured to detect a user's gaze using a first image that iscaptured at the time of irradiating a preset first pattern and a secondimage that is captured at the time of irradiating a preset secondpattern, and the processor detects a user's gaze by extractingreflection points of each of the first image and the second image andusing remaining reflection points excluding reflection points at a sameposition from among the extracted reflection points.

The processor may generate a difference image on the first image and thesecond image, extract a reflection point from the generated differenceimage, and detect a user's gaze using the reflection point that isextracted from the difference image.

The processor may extract feature points respectively from the firstimage and the second image, and if positions of the extracted featurepoints are different, correct the second image so that the position ofthe feature point extracted from the second image is the same as theposition of the feature point extracted from the first image.

The processor may, while the position of the feature point of the firstimage and the position of the feature point of the second image arecoincided, extract a center point of a pupil in an eye area of a userfrom each of the first image and the second image, and if the positionsof the extracted center points of the pupil are different, correct thesecond image to be coincided with the position of the extracted centerpoint of the pupil of the first image.

The inputter may capture an eye area of the user at speed of 30 to 60frames per second.

The light emitting unit may include a plurality of light emitting diodewhich are spaced apart from each other, and the processor may controlthe plurality of light emitting diode so that different light emittingdiode emit light according to the first pattern and the second pattern.

The processor may control the plurality of light emitting diode so thatall the plurality of light emitting diode emit light according to thefirst pattern and that all the plurality of light emitting diode not toemit light according to the second pattern.

The electronic device may further include a memory configured to storecoordinate information on remaining reflection points excludingreflection points at a same position, from among the extractedreflection points, and the processor may detect a user's gaze byextracting a plurality of reflection points from a third image that iscaptured at the time of irradiating of a third pattern, from among theplurality of images, and using a reflection point selected based on thestored coordinate information from among the extracted plurality ofreflection points.

The light emitting unit may include a plurality of light emitting diodewhich are spaced apart from each other, and the processor may controlthe plurality of light emitting diode so that all the plurality of lightemitting diode emit light according to the third pattern.

A controlling method of an electronic device includes alternatelyemitting light having a preset pattern to an eye area of a user;receiving a plurality of images which capture an eye area of the user;and extracting reflection points of each of a first image that iscaptured at the time of irradiating a preset first pattern and a secondimage that is captured at the time of irradiating a preset secondpattern; and detecting a user's gaze by extracting reflection points ofeach of the first image and the second image and using remainingreflection points excluding reflection points at a same position fromamong the extracted reflection points.

The detecting may include generating a difference image on the firstimage and the second image, extracting a reflection point from thegenerated difference image, and detecting a user's gaze using thereflection point that is extracted from the difference image.

The method may further include extracting feature points respectivelyfrom the first image and the second image; and if positions of theextracted feature points are different, correcting the second image sothat the position of the feature point extracted from the second imageis the same as the position of the feature point extracted from thefirst image.

The method may further include, while the position of the feature pointof the first image and the position of the feature point of the secondimage are coincided, extracting a center point of a pupil in an eye areaof a user from each of the first image and the second image; and if thepositions of the extracted center points of the pupil are different,correcting the second image to be coincided with the position of theextracted center point of the pupil of the first image.

The inputting may include capturing an eye area of the user at speed of30 to 60 frames per second.

The emitting may include emitting different light emitting diode fromamong a plurality of light emitting diode which are spaced apart fromeach other according to the first pattern and the second pattern.

The emitting may include emitting the plurality of light emitting diodeaccording to the first pattern and not emitting the plurality of lightemitting diode according to the second pattern.

The method may further include storing coordinate information onremaining reflection points excluding reflection points at a sameposition, from among the extracted reflection points; extracting aplurality of reflection points from a third image that is captured atthe time of irradiating of a third pattern from among the plurality ofimages; and detecting a user's gaze using reflection points that areselected based on the stored coordinate information from among theextracted plurality of reflection points.

The emitting may include emitting all the plurality of light emittingdiode that are spaced apart from each other according to the firstpattern and the second pattern.

According to an exemplary embodiment, a computer readable recordingmedium including a program for executing a control method of anelectronic device, wherein the control method may include alternatelyemitting light having a preset pattern to an eye area of a user;receiving a plurality of images which capture an eye area of the user;and extracting reflection points of each of a first image that iscaptured at the time of irradiating a preset first pattern and a secondimage that is captured at the time of irradiating a preset secondpattern; and detecting a user's gaze by extracting reflection points ofeach of the first image and the second image and using remainingreflection points excluding reflection points at a same position fromamong the extracted reflection points.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects of the present disclosure will be moreapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, in which like reference numerals referto like elements, and wherein:

FIG. 1 is a view to describe a method of detecting a user's gaze in anoutdoor environment according to an exemplary embodiment,

FIG. 2 is a block diagram to describe a brief configuration of anelectronic device according to an exemplary embodiment,

FIG. 3 is a block diagram to illustrate a configuration of theelectronic device of FIG. 2 in greater detail,

FIG. 4 is a view illustrating an operation of the light emitting unitaccording to an exemplary embodiment,

FIG. 5 is a view illustrating a difference image between an image thatis photographed according to an operation of the light emitting unit ofFIG. 4 and a captured image,

FIG. 6 is a view illustrating an operation of the light emitting unitaccording to another exemplary embodiment,

FIG. 7 is a view illustrating a difference image of an image that iscaptured according to an operation of the light emitting unit of FIG. 6and the captured image,

FIG. 8 is a view to describe a method of compensating a movementaccording to an exemplary embodiment,

FIG. 9 is a view illustrating a result that is deducted according to anexemplary embodiment,

FIG. 10 is a view to describe a method of detecting feature points froman image which is input thereafter, based on a result illustrated inFIG. 9,

FIG. 11 is a flowchart to describe a method of detecting a user's gazeaccording to an exemplary embodiment.

DETAILED DESCRIPTION

Certain example embodiments are described in greater detail below withreference to the accompanying drawings.

The terms used in the embodiments of the present disclosure are generalterms which are widely used now and selected considering the functionsof the present disclosure. However, the terms may vary depending on theintention of a person skilled in the art, a precedent, or the advent ofnew technology. In addition, in a special case, terms selected by theapplicant may be used. In this case, the meaning of the terms will beexplained in detail in the corresponding detailed descriptions.Accordingly, defining the terms used herein will be based on themeanings of the terms and overall contents of exemplary embodiments,rather than simple names of the terms.

As embodiments may have a variety of modifications and several examples,certain embodiments will be exemplified in the drawings and described indetail in the description thereof. However, this does not necessarilylimit the scope of the embodiments to a specific embodiment form.Instead, modifications, equivalents and replacements included in thedisclosed concept and technical scope of this specification may beemployed. While describing embodiments, if it is determined that thespecific description regarding a known technology obscures the gist ofthe disclosure, the specific description is omitted.

In the present disclosure, relational terms such as first and second,and the like, may be used to distinguish one entity from another entity,without necessarily implying any actual relationship or order betweensuch entities. In embodiments of the present disclosure, relationalterms such as first and second, and the like, may be used to distinguishone entity from another entity, without necessarily implying any actualrelationship or order between such entities.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.The terms “include”, “comprise”, “is configured to,” etc., of thedescription are used to indicate that there are features, numbers,steps, operations, elements, parts or combination thereof, and theyshould not exclude the possibilities of combination or addition of oneor more features, numbers, steps, operations, elements, parts or acombination thereof.

According to embodiments, a “module” or “unit” performs at least onefunction or operation, and may be implemented as hardware or software,or a combination of hardware and software. In addition, a plurality of‘modules’ or a plurality of ‘units’ may be integrated into at least onemodule and may be realized as at least one processor except for‘modules’ or ‘units’ that should be realized in a specific hardware.

Hereinbelow, certain embodiments will now be described in greater detailwith reference to the accompanying drawings to enable those skilled inthe art to work the same with ease. However, embodiments may be realizedin a variety of different configurations, and not limited todescriptions provided herein. Further, those that are irrelevant withthe description are omitted so as to describe embodiments more clearly,and similar drawing reference numerals are used for the similar elementsthroughout the description.

Hereinbelow, the present disclosure will be further described withreference to drawings.

FIG. 1 is a view to describe a method of detecting a user's gaze in anoutdoor environment according to an exemplary embodiment.

Referring to FIG. 1, an electronic device 100 may track a gaze of a user10 using a light emitting unit 110 and an inputter 120. To be specific,the electronic device 100 may make the light emitting unit 110 includinga plurality of light-emitting elements emit light. Then, the electronicdevice 100 may receive an image which captures a reflection point thatis a point that light is reflected from among an eye area of the user 10through the inputter 120 and track a gaze of the user 10 using the inputimage. Herein, the inputter 120 may be a capturing unit such as aninfrared camera which captures eyes of the user 10.

In the outdoor, external light 115 other than the light emitting unit110 exists. Here, the external light 115 may include sun light, streetlight, and the like. Accordingly, the eye image of the user 10 inputthrough the inputter 120 may include a reflection point of the externallight 115 other than the light emitting unit 110.

In this case, the electronic device 100 uses the remaining points exceptthe reflection points of the external light 115 among the plurality ofreflection points included in the eye image of the user 10 input throughthe inputter 120 to tract the gaze of the user 10. This will bedescribed below in detail with reference to FIGS. 2 to 11.

FIG. 2 is a block diagram to describe a brief configuration of anelectronic device according to an exemplary embodiment.

Referring to FIG. 2, the electronic device 100 includes the lightemitting unit 110, the inputter 120, and a processor 130.

The light emitting unit 110 alternately irradiates light having a presetpattern. Specifically, the light emitting unit 110 includes a pluralityof light emitting diode spaced apart from each other for irradiatinglight to a user's eye area, and different light emitting diode may emitlight according to a preset pattern under the control of the processor130.

For example, if the light emitting unit 110 includes four light emittingdevices arranged in a rectangular shape and is referred to as lightemitting diode 1, 2, 3, 4 in the clockwise direction from the upper leftlight emitting element, the second pattern in which the light emittingdiode 3 and 4 emit light can be repeated after the first pattern inwhich the light emitting diode 1 and 2 emit light according to a presetpattern. Here, the light emitting element that emits light according tothe first pattern and the light emitting element that emits lightaccording to the second pattern may be different light emitting diode.In the meantime, the light emitting element includes an LED such as anincandescent light, an infrared ray light, an LED light and the like,and the type and the number thereof are irrelevant.

Meanwhile, the light emitting unit 110 can change the pattern at apreset speed under the control of the processor 130 and irradiate light.Here, the preset speed may correspond to the photographing speed of theimage sensing unit which photographs the eye area of a user. However,the present disclosure is not limited thereto, and the light emittingunit 110 may change the pattern at a slower or faster speed than thephotographing speed of the image capturing unit.

The inputter 120 receives a plurality of images of a user's eye area.Here, the plurality of received images may include reflection points bythe light emitting unit 110 and external lighting. Specifically, theinputter 120 may include a capturing unit for imaging the user's eyearea. Here, the inputter 120 includes a general camera using a visibleray, an infrared camera, and the like, and the kind thereof isirrelevant. In this case, the inputter 120 may include a camera forphotographing at a speed of 20 to 60 frames per second in addition to ahigh-speed camera for photographing at a speed of 2000 frames persecond.

In the meantime, when the electronic device 100 does not include animage capturing unit, the inputter 120 may include a communication unitthat receives a plurality of images of an eye area of the user from anexternal device. Here, the external device may be an apparatus having animage capturing unit for photographing the eye area of the user.

The processor 130 may first control the light emitting unit 110 toalternately emit light in a preset pattern. Here, the light emittingunit 110 may include a plurality of light emitting devices, and theprocessor 130 may control the light emitting unit 110 such that lightforming each pattern is irradiated by different light emitting diode.

The processor 130 can detect the user's gaze using the plurality ofimages input by the inputter 120. Here, the processor 130 can detect theuser's gaze using the first image captured at the irradiation point ofthe preset first pattern and the second image captured at theirradiation point of the second pattern among the plurality of inputimages. Here, the first image and the second image may be consecutiveframes. The first pattern and the second pattern may be formed byemitting light from different light emitting diode among a plurality oflight emitting diode constituting the light emitting unit 110.Accordingly, the user's eye area of the first image and the second imagemay include reflection points of different patterns.

Specifically, the processor 130 may extract the reflection points ofeach of the first image and the second image, and use the remainingreflection points excluding the reflection points contained in the samepositions of the first image and the second image among the extractedreflection points to detect user's gaze.

Specifically, the processor 130 may generate a difference image of thefirst image and the second image, and extract a reflection point fromthe generated difference image. Here, the first pattern and the secondpattern may be displayed on the difference images of the first image andthe second image because the first and second patterns are differentfrom each other. Here, since the reflection points by the externallighting are displayed at the same positions of the first image and thesecond image, the reflection points can be removed in the differenceimages of the first image and the second image. Accordingly, in thereflection points extracted from the difference images of the firstimage and the second image by the processor 130, the reflection pointsby external lighting are removed, and the processor 130 can use all thereflection points extracted from the difference images to detect theuser's gaze.

The processor 130 may extract the reflection points in each of the firstimage and the second image. Specifically, since the reflection point dueto light will have the greatest brightness, the processor 130 canextract a point having a high brightness as a reflection point. Theprocessor 130 may use the coordinates of the reflection points extractedfrom the first image and the second image to detect the user's line ofsight by excluding the reflection points having the same coordinates andusing the remaining reflection points. As described above, since thereflection points of different patterns are included in each image, itis possible to eliminate the reflection points due to external lightingand precisely detect the user's gaze without providing prior informationon the position of the light emitted from the light emitting unit 110.

When there is a position change of the user between photographing of thefirst image and the second image, the processor 130 corrects positionchange to extract reflection points that are required to detect theuser's gaze.

Specifically, the processor 130 can extract feature points from thefirst image photographed at the time of irradiating the light of thefirst pattern and the second image photographed at the time ofirradiating the light of the second pattern. Here, the feature point maybe separated from the reflection points included in the eye area of theuser. For example, it may be a starting point or a tail point of theuser's eye, or a point area included in an area other than the eye areaamong the user's face area, which is a characteristic that the structuredoes not change even if the position of the user's face moves. Here, ifonly the position of the face of the user is moved and the position ofthe eye is not moved, the processor 130 may extract the center point ofthe pupil as a feature point.

When the first pattern is formed by not emitting all of the plurality oflight emitting diode and the second pattern is formed by emitting allthe plurality of light emitting diode of the light emitting unit 110,the first image photographed at the time of irradiation of light in thefirst pattern may include reflection points due to external lighting. Inaddition, even if the position of the user's face moves, the positionshift of the reflection point due to the inter-frame external lightingwill not be great. Accordingly, the processor 130 may extract thereflection points due to the external lighting included in the firstimage as feature points, and the reflection points at positions closestto the feature points extracted from the first image, from among theplurality of reflection points included in the second image, as thefeature points.

The processor 130, if the position of the feature point extracted fromthe first image is different from the position of the feature pointextracted from the second image, may correct the second image so thatthe feature point of the second image is coincided with the featurepoint of the first image. Specifically, the processor 130 can move theposition of the entire second image such that the feature pointsextracted from the second image are located at positions of the featurepoints extracted from the first image.

In addition, when the direction of the gaze as well as the position ofthe user's face between the first image and the second image is changed,the processor 130 may correct the position of the face and the directionof the gaze to extract a reflection point necessary for detecting thegaze.

Specifically, through the above-described correction, in a state wherethe positional shift of the user's face is corrected, the processor 130can extract the center point of the pupil in the eye area of the user ofthe first image and the second image. Here, since the center point ofthe pupil is the lowest brightness area, the processor 130 can extractthe area having the lowest brightness in the image as the center pointof the pupil.

Specifically, the processor 130 may extract the center point of thepupil in the first image and the second image with the positions of thefeature points of the first image and the second image matched, and ifthe center points of the extracted pupils are different, the processormay correct the second image so that the center point of the secondimage coincides with the position of the center point of the pupilextracted from the first image. Specifically, the processor 130 cancorrect the second image such that the center point of the pupilextracted from the second image is located at the position of the centerpoint of the pupil extracted from the first image. At this time, theprocessor 130 may use an optical relationship between the center pointof the pupil and the reflection point for compensating for eye movement.Specifically, since there is a constant ratio between the center pointof the pupil and the change of the position of the reflection point whenthe eye rotates, the processor 130 can correct the positional change ofthe reflection point using the change in the position of the centerpoint of the pupil.

Such a correction operation may be performed based on extracted featurepoints by extracting the feature points, after extracting all thereflection points of the first image and the second image. In themeantime, the feature points may be extracted and corrected beforeextracting the reflection points in each image, and then the reflectionpoints may be extracted from the first image and the corrected secondimage, respectively. Then, the processor 130 can detect the user's gazeusing the corrected reflection point.

The processor 130 can track the movement of the user's gaze whilerepeatedly performing the gaze detection. In addition, the processor 130may perform an event corresponding to the detected user's gaze or thegaze movement of the tracked user. For example, the processor 130 mayplace a cursor at a position viewed by the user through the gazedetection of the user, and may perform an operation of moving the cursoraccording to the movement of the user's gaze.

As described above, by using two images including reflection points forlight having different patterns, it is possible to grasp the pattern ofthe light emitting unit without prior knowledge of the pattern of thelight emitting unit, and it is possible to detect the reflect point todetect the user's gaze more correctly using a low-cost general cameraeven in an outdoor environment.

FIG. 3 is a block diagram to illustrate a configuration of theelectronic device of FIG. 2 in greater detail.

Referring to FIG. 3, the electronic device 100 may include the lightemitting unit 110, the inputter 120, a processor 130, a memory 140, acommunication unit 150, a video processor 160, and a display 170. Here,some operations of the light emitting unit 110, the inputter 120, andthe processor 130 are the same as those shown in FIG. 1, and redundantdescriptions are omitted.

The processor 130 may include a RAM 131, a ROM 132, a CPU 133, a GraphicProcessing Unit (GPU) 134, and a bus 135. The RAM 131, the ROM 132, theCPU 133, the Graphic Processing Unit (GPU) 134, and the like may beconnected to each other via the bus 135.

The CPU 133 accesses the memory 140 and performs booting using the O/Sstored in the memory 140. Then, the CPU performs various operationsusing various programs, contents, data, etc. stored in the memory 140.

The ROM 132 stores a command set for booting the system. When theturn-on command is input and power is supplied, the CPU 133 copies theO/S stored in the memory 180 to the RAM 131 according to the instructionstored in the ROM 132, executes O/S to boot the system. When the bootingis completed, the CPU 133 copies various programs stored in the memory140 to the RAM 131, executes the program copied to the RAM 131, andperforms various operations.

The GPU 134 displays the UI on the display 170 when the booting of theelectronic device 100 is completed. Specifically, the GPU 134 cangenerate a screen including various objects such as an icon, an image,and a text using an operation unit (not shown) and a rendering unit (notshown). The operation unit calculates attribute values such ascoordinate values, shapes, sizes, and colors to be displayed by therespective objects according to the layout of the screen. The renderingunit generates screens of various layouts including the objects based onthe attribute values calculated by the operation unit. The screen (oruser interface window) generated by the rendering unit is provided to adisplay, and is displayed in the main display area and the sub displayarea, respectively. In the above description, the image processingoperation according to the present disclosure is performed by the GPU134. However, in actual implementation, the image processing operationcan be performed by the CPU 133 or the GPU 134.

The memory 140 may be implemented as a storage medium in the electronicdevice 100 and an external storage medium, for example, a removable diskincluding a USB memory, a web server through a network, and the like. Tobe specific, the memory 140 may include a hard disk, an SSD, a memorycard, a ROM, a USB memory.

Specifically, the memory 140 may store the coordinates of the reflectedpoints extracted using the plurality of images input through theinputter 120. Here, the coordinates of the stored reflection points canbe used for the user's gaze detection. The coordinates of the reflectedpoints to be stored may be coordinates excluding the coordinates of thereflection points at the same position included in the plurality ofimages among the plurality of reflection points extracted from theplurality of images.

The processor 130, using the first image photographed at the time ofirradiating light of the first pattern and the second image photographedat the time of irradiating light of the second pattern, from among theplurality of input images, may store coordinates of reflection pointsextracted to use for detection of the user's gaze in the memory 140. Theprocessor 130, when coordinates of the reflection points to be used fordetection of the user's gaze are determined, may control the lightemitting unit 110 to irradiate light with the third pattern that isdistinguished from the first pattern and the second pattern.

For example, if the first pattern is formed by emitting the first andthird light emitting diode among the four light emitting diode and thesecond pattern is formed by emitting the second and fourth lightemitting diode, the third pattern can be formed by making all the fourlight emitting diode emit light. This is because the external light isremoved by using the first pattern and the second pattern, and thepattern necessary for detecting the user's gaze is determined, and afterthe necessary pattern is determined, the processor 130 may remove thereflection point caused by the external light even if all the lightemitting diode are made to emit light.

Specifically, the processor 130 may extract a plurality of reflectionpoints from the third image photographed at the time of irradiatinglight of the third pattern among the plurality of images input by theinputter 120. Here, the extracted reflection point may include both thereflection point by the light emitting unit 110 and the reflection pointby the external lighting.

The processor 130 may then select a reflection point to be used fordetecting user's gaze based on the coordinate information of thereflection points stored in the memory 140 among the reflection pointsextracted from the third image. At this time, the processor 130 canselect the reflection point most similar to the coordinate informationof the stored reflection point among the reflection points extractedfrom the third image. Then, the processor 130 can detect the user's gazeusing the selected reflection point. That is, a reflection pointnecessary for user's gaze detection can be detected using a pattern ofthe reflection points determined in the frame after the pattern ofreflection points necessary for user's gaze detection is determined.Accordingly, it is not necessary to generate a difference image, and thememory and time required for detecting the user's gaze are reduced. Inthe meantime, a method of detecting a reflection point necessary foruser's gaze detection using the determined pattern of reflection pointswill be described in detail with reference to FIGS. 9 and 10.

The communication unit 150 is configured to perform communication withvarious types of external devices according to various types ofcommunication methods. Specifically, the communication unit 150 canperform communication with an external device such as an externalimaging device, an external server, or an external display device.

Specifically, the communication unit 150 may receive a plurality ofimages from an external device through a wired system such as anantenna, a cable, or a port, or receive a plurality of images through awireless system such as Wi-Fi and Bluetooth.

The communication unit 150 may transmit information on user's gazedetection that is calculated by the operation of the processor 130 to anexternal device.

In FIG. 3, the inputter 120 and the communication unit 150 are shown asseparate components. However, the electronic device 100 may not includean image capturing unit, or may be configured such that the inputter 120may include a part or whole of a function of the communication unit 150.

The video processor 160 is a component for processing the contentreceived through the communication unit 150 or the video data includedin the content stored in the memory 140 when the display 170 is providedin the electronic device 100. The video processor 160 may performvarious image processing such as decoding, scaling, noise filtering,frame rate conversion, resolution conversion, etc. on the video data.

The video processor 160 may then perform image processing on the videodata based on the user's gaze detected by the processor 130.Specifically, the video processor 160 may change the area to bedisplayed on the display 170 among the received images based on thedetected user's gaze, or may display a cursor on the received image.

The display 170 can display an image corresponding to the detected gazeof the user. Specifically, if the user is gazing at an image displayedon the display 170, the display 170 may further display a cursor or thelike at a position corresponding to the user's gaze of the displayedimage. In the meantime, if the user is gazing at the surrounding space,the display 170 may display an image of the peripheral space under thecontrol of the processor 130, and may further display a cursor or thelike at a position corresponding to the user's gaze of the displayedimage.

Meanwhile, although not shown, the electronic device 100 may furtherinclude a sensor such as an illuminance sensor to determine whether thespace in which the electronic device 100 is located is indoor oroutdoor. At this time, if it is determined that the space in which theelectronic device 100 is located is the room according to the sensedillumination value, the processor 130 can emit all of the plurality oflight emitting devices constituting the light emitting unit 110. This isbecause the input image is less likely to include reflection points dueto external lighting in the case of the indoor. However, even if it isdetermined that the electronic device 100 is located indoors, theprocessor 130 may control the light emitting unit 110 to alternatelyirradiate light in different patterns.

FIG. 4 is a view illustrating an operation of the light emitting unitaccording to an exemplary embodiment.

The electronic device can control the light emitting unit to alternatelyirradiate light of different patterns as shown in FIGS. 4A and 4B.Referring to FIG. 4, the light emitting unit may include four lightemitting diode. For convenience of description, the light emittingelement on the upper left side is numbered 1, the light emitting deviceon the upper right side is numbered 2, the light emitting element at thelower right end is referred to as 3, and the light emitting element atthe lower left end is referred to as 4 in an clockwise direction.

When the electronic device irradiates light in the first patternaccording to a preset pattern, the light emitting diode 2 and 4 may emitlight, and the light emitting diode 1 and 3 may not emit light, as shownin FIG. 4A. When the light is irradiated with the second pattern, asshown in FIG. 4B, the light emitting diode 1 and 3 may be emitted whilethe light emitting diode 2 and 4 may not be emitted.

The external lighting may keep emitting light irrelevantly to a patternof the light emitting unit.

In FIG. 4, a plurality of light emitting diode in a diagonal directionemit light alternately, but in actual implementation, the light emittingdevices emit light in the horizontal direction (1, 2/3, 1, 4 arealternately emitted) or in the vertical direction (1, 4/2, and 3 arealternately emitted). In addition, all of the light emitting diode mayemit light in the first pattern, and all of the light emitting diode maynot emit light in the second pattern.

In FIG. 4, a light emitting unit has four light emitting diode disposedin a quadrangle shape, but the number of light emitting units may bethree or less, or four or more. The disposition shape is not limited toquadrangle.

FIG. 5 is a view illustrating a difference image between an image thatis photographed according to an operation of the light emitting unit ofFIG. 4 and a captured image.

Referring to FIG. 5, FIG. 5A is a first image taken when light isirradiated in the first pattern of FIG. 4A, and FIG. 5B is a secondimage photographed when light is irradiated in the second pattern ofFIG. 4B. In addition, FIG. 5C is a difference image of the first imageand the second image. This is based on viewing the light emitting unitand the user in the same direction. When the light emitting unit and theuser are viewed in the opposite direction, FIG. 5A is an image takenwhen light is irradiated in the pattern of FIG. 4B, and FIG. 5B may be aphotographed image when light is irradiated in the pattern of FIG. 4A.Hereinafter, for convenience of description, the description will bemade with reference to the case of viewing the light emitting unit andthe user in the same direction.

Referring to FIG. 5A, the first image may include reflection points 511and 513 by the light emitting unit and a reflection point 512 byexternal lighting.

Referring to FIG. 5B, the second image may include the reflection points521, 523 by the light emitting unit which emits light in the secondpattern and the reflection point 522 by external lighting.

Referring to FIG. 5C, the reflection points 511 and 513 of the firstpattern and the reflection points 521 and 523 of the second pattern areformed by different light emitting diode, respectively. However, thereflection points 512 and 522 due to the external lighting continue toemit light regardless of the pattern of the external lighting, and maybe displayed at the same position as there is no movement of the userbetween the first image capturing and the second image capturing.Accordingly, the reflection points 512 and 522 due to the externallighting can be removed at the time of generation of the differenceimage.

Accordingly, the electronic device can extract only the reflectionpoints 511, 513, 521, and 523 due to the light emitting unit from whichthe reflection points 512 and 522 are removed by the external lighting,and may detect a user's gaze.

FIG. 6 is a view illustrating an operation of the light emitting unitaccording to another exemplary embodiment.

The electronic device can control the light emitting unit to alternatelyirradiate light of different patterns as shown in FIGS. 6A and 6B.Referring to FIG. 6, the light emitting unit may include four lightemitting diode. For convenience of description, the upper left lightemitting element is 1, the right upper light emitting element is 2, thelight emitting element at the lower right end is referred to as 3, andthe light emitting element at the lower left end is referred to as 4.

When the electronic device irradiates light in the first patternaccording to a preset pattern, the light emitting element may not emitlight as shown in FIG. 6A. When the light is irradiated with the secondpattern, as shown in FIG. 6B, all the light emitting diode can beemitted.

The external lighting 115 can keep emitting light regardless of apattern of the light emitting unit.

FIG. 7 is a view illustrating a difference image of an image that iscaptured according to an operation of the light emitting unit of FIG. 6and the captured image.

Referring to FIG. 7, FIG. 7A is a first image taken when light isirradiated in the first pattern of FIG. 6A, FIG. 7B is a first imagetaken when light is irradiated in the second pattern of FIG. 6B. Inaddition, FIG. 7C is a difference image of the first image and thesecond image. At this time, it is assumed that the user's positionalshift has occurred between the first image capturing and the secondimage capturing.

Referring to FIG. 7A, the first image has no light emitting elementwhich emits light according to the first pattern and thus may includeonly the reflection point 711 by external lighting.

Referring to FIG. 7B, in the second image, since all of the lightemitting diode emit light according to the second pattern, thereflection points 722, 723, 724, and 725 by the light emitting unit andthe reflection points 721 by external lighting can be included.

At this time, since the position of the user is shifted between thefirst image capturing and the second image capturing, even if theexternal lighting emits light at the same position, the position of thereflection point 711 due to external lighting in the first image, andthe position of the reflection point 721 due to the external lighting inthe second image may be different.

Referring to FIG. 7C, in the difference image of the first image and thesecond image, the reflection point 711 displayed in the first image andthe reflection points 721-725 displayed in the second image can bedisplayed.

As described above, when the position of the user is moved, thereflection point due to the external lighting cannot be removed only bygenerating the difference image, so that the electronic device canperform correction for the movement of the user. This will be describedbelow with reference to FIG. 8.

FIG. 8 is a view to describe a method of compensating a movementaccording to an exemplary embodiment. Hereinbelow, for convenience ofdescription, an image captured according to a pattern of the lightemitting unit as illustrated in FIG. 4 will be used as a reference.

Referring to FIG. 8, FIG. 8A is an image including a first pattern ofFIG. 4A and a reflection point displayed when light is irradiated byexternal lighting, FIG. 8B further illustrates reflection points in acase when light is irradiated by external light along with the secondpattern of FIG. 4B in addition to FIG. 8A. At this time, it is assumedthat the user has moved between the first image capturing and the secondimage capturing. FIG. 8C is an image in which the movement of the useris corrected. On the other hand, this is based on viewing the lightemitting portion and the user in the same direction, and in the casewhere the light emitting portion and the user are viewed in the oppositedirection, FIG. 8A can be a photographed image when light is irradiatedin the pattern of FIG. 4B. Hereinafter, for convenience of description,the description will be made with reference to the case of viewing thelight emitting portion and the user in the same direction.

Referring to FIG. 8A, the first image includes a plurality of reflectionpoints 811 that are generated when light of the first pattern isirradiated outdoor, and the plurality of reflection points 811 mayinclude the reflection point 812 by external lighting.

Referring to FIG. 8B, the second image includes a plurality ofreflection points 821 that are generated when light of a second patternis emitted outdoors, and the plurality of reflection points 821 includereflection points 822 by the external lighting. A plurality ofreflection points 811 corresponding to the first pattern shown in FIG.8B are shown for describing the movement of the user, and are notreflection points displayed in the actual image.

Here, since the position of the user is shifted between the first imagecapturing and the second image capturing, even if the external lightingis emitted at the same position, the position of the reflection point812 by the external lighting in the first image may be different fromthe position of the reflection point 822 due to the external lighting inthe two images.

The electronic device may extract the feature point that isdistinguished from the reflection point included in the eye area of theuser in the first image captured at the time of irradiating the light ofthe first pattern and the second image captured at the time ofirradiating the light of the second pattern and correct the positionmovement. For example, it may be a starting point or a tail point of theuser's eye, or a point area included in an area other than the eye areaamong the user's face area, which is a characteristic that the structuredoes not change even if the position of the user's face moves. Here, ifonly the position of the face of the user is moved and the position ofthe eye is not moved, the processor 130 may extract the center point ofthe pupil as a feature point.

Specifically, the electronic device can calculate the positionalmovement of the user using the extracted feature points on each image,and correct the position of the second image based on the calculatedvalue.

Accordingly, as shown in FIG. 8C, the positions of the reflection points812 and 832 by the external lighting can be matched through thecorrection. Then, the electronic device can detect the user's gaze usingthe reflection point excluding the reflection points 812 and 832 due toexternal lighting among the plurality of reflection points 811 and 831by using the position coordinates of the corrected reflection point.

FIG. 9 is a view illustrating a result that is deducted according to anexemplary embodiment.

The electronic device can distinguish the reflection points of theexternal lighting and the reflection points of the light emitting unitamong a plurality of reflection points included in the image of theuser's eye area through the above-described process. In addition, theelectronic device can store information about the reflection point bythe light emitting unit. Here, the stored information may be thecoordinate information of the reflection point by the light emittingunit.

Referring to FIG. 9, the image which captures the eye area displays aplurality of reflection points including the reflection point 911 by theexternal lighting and the reflection point 921 by the light emittingunit. Here, the reflection point 921 by the light emitting unit may bein a rectangular shape.

FIG. 10 is a view to describe a method of detecting feature points froman image which is input thereafter, based on a result illustrated inFIG. 9.

The electronic device can control the light emitting unit to emit aplurality of light emitting diode without a pattern by dividing thereflection point by the light emitting unit and storing the coordinateinformation of the reflection point by the light emitting unit. This isbecause it is possible to eliminate the reflection points for theexternal lighting even if the light is not irradiated according to thepattern using the stored coordinate information.

Specifically, the electronic device can receive a plurality of mages ofthe eye area of the user while emitting a plurality of light emittingdiode, and extract a plurality of reflection points from the receivedimage. Then, the electronic device can compare the various candidatesand the stored coordinate information at the extracted plurality ofreflection points, and select the reflection points included in thecandidate most similar to the stored coordinate information. Theelectronic device can then detect the user's gaze using the selectedreflection point.

Here, the electronic device can select a reflection point having aposition closest to the coordinate information of the stored pluralityof reflection points, or a reflection point having the most similarshape when the candidate reflection points are connected. When selectinga reflection point with a similar connection shape, the electronicdevice may take into account the distance and angle when connecting withthe adjacent reflection point.

Referring to FIGS. 10A to 10D, the electronic device may compare variouscandidates 1001, 1002, 1003, and 1004 from among a plurality ofreflection points and stored coordinate information on the reflectionpoint illustrated in FIG. 9 and may select the reflection point includedin the candidate 1004 which is most similar to the stored coordinateinformation.

FIG. 11 is a flowchart to describe a method of detecting a user's gazeaccording to an exemplary embodiment.

Referring to FIG. 11, first, the electronic device can alternatelysearch for the eyes having the preset pattern in the eye region of theuser in step S1110. Specifically, the electronic device can irradiatelight by controlling the light emitting unit including a plurality oflight emitting diode. Here, the preset pattern includes a plurality ofpatterns, and each pattern may be formed of light emitted from differentlight emitting diode among the plurality of light emitting diode.

The electronic device can receive a plurality of images of the user'seye area in step S1120. Specifically, the electronic device can capturean eye area of a user using the provided image capturing unit, orreceive an image photographed by an external device. Here, when theelectronic device is provided with the image capturing unit, the imagecapturing unit may be a general camera which photographs at a speed of30 frames to 60 frames per second.

Then, the electronic device can extract the reflection points of each ofthe first image and the second image in step S1130. Specifically, theelectronic device may extract a reflection point of each of thephotographed second images when light of the first pattern and thesecond pattern are irradiated when light of the first pattern among theplurality of input images is irradiated. Here, the first image and thesecond image may be consecutive frames.

Then, the electronic device can detect the user's gaze using theremaining reflection points except for the reflection points at the sameposition among the reflection points extracted from the respectiveimages in step S1140. Specifically, the electronic device can remove thereflection points at the same position using the difference image of thefirst image and the second image. If there is a movement of the userbetween the first image capturing and the second image capturing, theelectronic device can correct the image according to the movement of theposition and then detect the user's gaze using the reflection point.

As described above, by using two images which include reflection pointson light of different patterns, a pattern of a light emitting unit canbe grasped without prior information on the pattern of the lightemitting unit, and even in outdoor environment, a reflection point fordetecting a user's gaze correctly can be detected using a low-costgeneral camera.

Meanwhile, the various embodiments described above can be implemented ina recording medium that can be read by a computer or similar deviceusing software, hardware, or a combination thereof. In accordance with ahardware implementation, the embodiments described in this disclosuremay be implemented in the fields of application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs) fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, and an electrical unit forperforming other functions. In some cases, embodiments described hereinmay be implemented by processor 130 itself. According to a softwareimplementation, embodiments such as the procedures and functionsdescribed herein may be implemented in separate software modules. Eachof the software modules may perform one or more of the functions andoperations described herein.

A controlling method of the electronic device according to variousexemplary embodiments of the aforementioned present disclosure can bestored in a non-transitory readable medium. The non-transitory readablemedium can be mounted in various devices and used.

The non-transitory computer readable medium refers to a medium thatstores data semi-permanently, and is readable by an apparatus.Specifically, the above-described various applications or programs maybe stored in the non-transitory computer readable medium such as acompact disc (CD), a digital versatile disk (DVD), a hard disk, aBlu-ray disk, a universal serial bus (USB), a memory card, a ROM oretc., and may be provided.

The foregoing example embodiments and advantages are merely examples andare not to be construed as limiting. The present teaching can be readilyapplied to other types of apparatuses. Also, the description of theexample embodiments is intended to be illustrative, and not to limit thescope of the claims, and many alternatives, modifications, andvariations will be apparent to those skilled in the art.

What is claimed is:
 1. An electronic device comprising: a light emittingunit configured to alternately emit light having a preset pattern to aneye area of a user; an inputter configured to receive a plurality ofimages which capture the eye area of the user; a memory configured tostore coordinate information on remaining reflection points excludingreflection points at a same position, from among extracted reflectionpoints; and a processor configured to: obtain a first image that iscaptured at a time of irradiating a first pattern and a second imagethat is captured at a time of irradiating a second pattern, extractreflection points of each of the first image and the second image,detect a user's gaze by using remaining reflection points excludingreflection points at a same position from among the extracted reflectionpoints, detect the user's gaze by extracting a plurality of reflectionpoints from a third image that is captured at a time of irradiating of athird pattern, from among the plurality of images, and use a reflectionpoint selected based on the stored coordinate information from among theextracted plurality of reflection points.
 2. The electronic device ofclaim 1, wherein the processor is further configured to: generate adifference image on the first image and the second image, extract areflection point from the difference image, and detect the user's gazeusing the reflection point that is extracted from the difference image.3. The electronic device of claim 1, wherein the processor is furtherconfigured to: extract feature points respectively from the first imageand the second image, and if positions of the extracted feature pointsare different, correct the second image so that the position of thefeature point extracted from the second image is the same as theposition of the feature point extracted from the first image.
 4. Theelectronic device of claim 3, wherein the processor, while the positionof the feature point of the first image and the position of the featurepoint of the second image are coincided, is further configured to:extract a center point of a pupil in the eye area of the user from eachof the first image and the second image, and if the positions of theextracted center points of the pupil are different, correct the secondimage to be coincided with the position of the extracted center point ofthe pupil of the first image.
 5. The electronic device of claim 1,wherein the inputter captures the eye area of the user at speed of 30 to60 frames per second.
 6. The electronic device of claim 1, wherein thelight emitting unit comprises a plurality of light emitting diodes whichare spaced apart from each other, and wherein the processor controls theplurality of light emitting diodes so that different light emittingdiodes emit light according to the first pattern and the second pattern.7. The electronic device of claim 6, wherein the processor is furtherconfigured to control the plurality of light emitting diodes so that allthe plurality of light emitting diodes emit light according to the firstpattern, and all the plurality of light emitting diodes do not emitlight according to the second pattern.
 8. The electronic device of claim1, wherein the light emitting unit comprises a plurality of lightemitting diodes which are spaced apart from each other, and wherein theprocessor is further configured to control the plurality of lightemitting diodes so that all the plurality of light emitting diodes emitlight according to the third pattern.
 9. A controlling method of anelectronic device, the method comprising: alternately emitting lighthaving a preset pattern to an eye area of a user; obtaining a firstimage that is captured the eye area at a time of irradiating a firstpattern and a second image that is captured the eye area at a time ofirradiating a second pattern; extracting reflection points of each of afirst image and a second image; detecting a user's gaze by usingremaining reflection points excluding reflection points at a sameposition from among the extracted reflection points; storing coordinateinformation on remaining reflection points excluding reflection pointsat a same position, from among the extracted reflection points;extracting a plurality of reflection points from a third image that iscaptured at a time of irradiating of a third pattern from among aplurality of images; and detecting the user's gaze using reflectionpoints that are selected based on the stored coordinate information fromamong the extracted plurality of reflection points.
 10. The method ofclaim 9, wherein the detecting of the user's gaze comprises: generatinga difference image on the first image and the second image, extracting areflection point from the difference image, and detecting the user'sgaze using the reflection point that is extracted from the differenceimage.
 11. The method of claim 9, further comprising: extracting featurepoints respectively from the first image and the second image; and ifpositions of the extracted feature points are different, correcting thesecond image so that the position of the feature point extracted fromthe second image is the same as the position of the feature pointextracted from the first image.
 12. The method of claim 11, furthercomprising: while the position of the feature point of the first imageand the position of the feature point of the second image are coincided,extracting a center point of a pupil in the eye area of the user fromeach of the first image and the second image; and if the positions ofthe extracted center points of the pupil are different, correcting thesecond image to be coincided with the position of the extracted centerpoint of the pupil of the first image.
 13. The method of claim 9,further comprising capturing the eye area of the user at speed of 30 to60 frames per second.
 14. The method of claim 9, wherein the emitting oflight comprises emitting different light emitting diodes from among aplurality of light emitting diodes which are spaced apart from eachother according to the first pattern and the second pattern.
 15. Themethod of claim 14, wherein the emitting of light comprises: emittingthe plurality of light emitting diodes according to the first pattern,and not emitting the plurality of light emitting diodes according to thesecond pattern.
 16. The method of claim 9, the emitting of lightcomprises emitting all the plurality of light emitting diodes that arespaced apart from each other according to the first pattern and thesecond pattern.
 17. A computer readable recording medium including aprogram for executing a control method of an electronic device, whereinthe control method comprises: alternately emitting light having a presetpattern to an eye area of a user; obtaining a first image that iscaptured the eye area at a time of irradiating a first pattern and asecond image that is captured the eye area at a time of irradiating asecond pattern; extracting reflection points of each of a first imageand a second image; detecting a user's gaze by using remainingreflection points excluding reflection points at a same position fromamong the extracted reflection points; storing coordinate information onremaining reflection points excluding reflection points at a sameposition, from among the extracted reflection points; extracting aplurality of reflection points from a third image that is captured at atime of irradiating of a third pattern from among a plurality of images;and detecting the user's gaze using reflection points that are selectedbased on the stored coordinate information from among the extractedplurality of reflection points.